Tested with OpenLLaMA 3bv2-q8_0
The score seems to be much lower than master. For reference the HF final (10-shot) score is 71.6 on the F16 model.

We might get a speedup and similar scores as master by using a whole task in one ctx window. And inserting BOS/EOS in between the full queries. Something like [BOS] context+ending0 [EOS] [BOS] context+ending1 ... I think one whole task should fit a ctx of 1024. If this works for one task, we could possibly include an additional task for even more speedup.

The score seems to be much lower than master. For reference the HF final (10-shot) score is 71.6 on the F16 model.

Hm, not sure. I tested with LLaMA-2 7B and the result matches 100% for the 800 tasks I checked. At 400 tasks this PR and master are both 0.25 higher compared to what is posted in the table in #2321. My hypothesis was that perhaps this is due to eps_rmse (I'm using 5e-6 instead of 1e-5 as it gives slightly lower perplexity).

It may be something with the openllama models. I will run a test using llama2-7b.

Here is what I get with Master and this PR using OpenLLaMA-3B and fp16. You only see one curve because the result of both runs is identical.

Yes, my bad. I was comparing the PR against a run I did when the hellaswag was implemented. This means that some changes since then have lowered the scores that much!

And here is a quick comparison between 0-shot and 3-shot HellaSwag using OpenLLaMA-3B and fp16. 3-shot is 1.375 higher after 800 tasks (but the statistical uncertainty of ~0.9 of this estimate is still much too high to be able to tell for sure by how much 3-shot improves vs 0-shot).

And here is a quick comparison between 0-shot and 3-shot

How does the 3shot work? By using EOS BOS in between the example tasks?

And here is a quick comparison between 0-shot and 3-shot

How does the 3shot work? By using EOS BOS in between the example tasks?

I tried a bunch of different stuff. The best approach so far is illustrated by the first task of the dataset:

Roof shingle removal: <item>A screen appears with a layout of a home and a lot of black, white and red words that say "tool box buzz news, reviews and information product reviews by industry expert todd fratzel". a picture of a tool appears on a red screen and to the right there are black words that say " ridgid model r040sca roofing cutter ".</item><item>This video allows viewers to hear the testimony of a customer who has used the pro roofing service. First the man tells how he heard about the service and how great it is. he also says that he chose pro roofing because the neighbors chose it and persuaded him by telling how great the service is.</item><item>A white screen appears with a black brand name that say's "the shingle hog" and it has a picture of a hog in the o of the word hog. a quick picture of a red and black tool is shown, then a man is talking and clips of people removing shingles play in between moments shown when the man talks.</item><item>A man is sitting on a roof. he
3
is using wrap to wrap a pair of skis.</item>
is ripping level tiles off.</item> 
is holding a rubik's cube.</item>
starts pulling up roofing on a roof.</item>

The first 3 query+response surrounded by <item>...</item> are taken from the HellaSwag training dataset, the last is the actual task query from the validation dataset. Not adding some kind of a separator between the sentences does not improve the score. This kind of makes sense as the sentences are not necessarily logically associated with each other, even if they come from the same activity group. I tried a few variations (e.g., <p>, <li>, etc., or making a json of the type "query":"response", or using [context]: [response]:, etc.), and the above version produces the highest score on 7B LLaMA-2 (0.95 higher than 0-shot after 10000 tasks).

I could not go beyond 3 because I'm running into a ggml_new_object: not enough space in the context's memory pool error despite using -c 1024 and the context being just 647 tokens when it fails

This could happen if you try to pass more tokens than n_batch to llama_eval. If you need to evaluate more than 512 tokens, you would have to split the input into multiple calls to llama_eval.

I could not go beyond 3 because I'm running into a ggml_new_object: not enough space in the context's memory pool error despite using -c 1024 and the context being just 647 tokens when it fails

This could happen if you try to pass more tokens than n_batch to llama_eval.

With other words, I need to split into batches if the number of tokens is greater than n_batch? Or can one now use n_batch > 512 without running into issue (as I did last time I tried that).

Set the n_batch = n_ctxif this is the problem.

I need to split into batches if the number of tokens is greater than n_batch

Yes, that's correct

Or can one now use n_batch > 512 without running into issue (as I did last time I tried that).

This looks like CUDA, which still uses the scratch buffers, which have a maximum batch size of 512. With the other backends, you should be able to use any batch size. However, keep in mind that the command line parsing at common.cpp caps the batch size to 512 regardless, so while it may be possible to initialize llama.cpp with a higher batch size in some backends, you need to specify it manually in llama_context_params.

@slaren Yes, I'm running on CUDA (doing HellaSwag on the CPU is a hopeless undertaking). But is it really the CUDA backend?. The error is raised in ggml_new_object(), which is called from ggml_new_tensor_2d() before things even went into the CUDA backend.

The first 3 query+response surrounded by ... are taken from the HellaSwag training dataset,

Are you saying you do the evaluation by first "steering" the model using the training dataset?

Are you saying you do the evaluation by first "steering" the model using the training dataset?

If this is how you want to see it, sure. The way I see it, this is basically my interpretation of what few-shot evaluation is.

The error is raised in ggml_new_object(), which is called from ggml_new_tensor_2d() before things even went into the CUDA backend.

So the problem is that due to some limitations, the CUDA backend still uses ggml scratch buffers for memory allocation, while the other backends have already moved to the graph allocator. The size of the scratch buffers was determined manually for a batch size of 512, while the graph allocator calculates the size of buffers dynamically and therefore can use any batch size. When using the graph allocator, the compute context is no_alloc, which means that tensor data is not allocated during calls such as ggml_new_tensor_2d(), but rather it is done later in a call to ggml_allocr_alloc_graph.

Are you saying you do the evaluation by first "steering" the model using the training dataset?

If this is how you want to see it, sure. The way I see it, this is basically my interpretation of what few-shot evaluation is.

Yes, but I do not think the training dataset should be used at all when testing. Have you tested using only data from the validation dataset?

So the problem is that due to some limitations, the CUDA backend still uses ggml scratch buffers for memory allocation

So, I guess the long-term solution would be to move the CUDA backend to the graph allocator. But what is a quick fix? Just play around with MEM_REQ_SCRATCH0 and MEM_REQ_SCRATCH1 until it runs? Or set n_batch and hope for the best? As far as I can tell increasing n_batch will not help as it runs out of space in the second scratch buffer, which does not depend on context or batch size.

But what is a quick fix? Just play around with MEM_REQ_SCRATCH0 and MEM_REQ_SCRATCH1 until it runs?

That should work. However, I think that the proper solution would be to respect n_batch in the perplexity tool, do the evaluation in multiple calls if needed.

I noticed that I had not downloaded OpenLLaMA-v2, so the above graph is for OpenLLaMA-v1. For the sake of completeness, here is a graph showing HellaSwag scores for OpenLLaMA-v2-3B, this time for all 10042 tasks. 3-shot ends with a score of 70.8, which is not too far from the 10-shot 71.6 result posted on HF.

Maybe test this PR in the GGUF branch. The perplexity is slightly lower with the changes there, so hellaswag scores could be higher.